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ABSTRACT 

As online degrees and programs increase in number and popularity so does the need for 
excellence and quality in the programs and courses offered. Becoming more scholarly in 
online course delivery, especially in the evaluation or assessment of those classes is 
essential for teaching and learning. 

This paper explores the pilot of an evaluation of student learning through anonymous 
feedback at mid-semester in seven online courses. Electronic Formative Assessment of 
Classroom Teaching (eFACT) is a process of gathering anonymous student feedback 
through a faculty consultant using e-mail. This process gives the online instructor the 
opportunity to make changes to the delivery of the course while the class is in session. 
Instructors felt they gained useful and meaningful information and were able to make 
changes in their delivery format midway through the semester. Student learning was 
helped through the use of online features that made the class more social and 
interactive. Perceived or real communication issues with the instructor and classmates 
hindered student learning. Although often citing the "nature" of online learning, the 
social aspect of learning seemed to be missing for many students. Assessments measures 
like eFACT can elicit detailed perceptions of student learning while the class is in session. 
It can affect the quality of the delivery method of the course by giving instructors 
immediate feedback as students reflect on their learning midway through the course. 
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INTRODUCTION 

Online degrees and programs continue to increase in number and popularity around the 
world. According to the National Center of Education Statistics (2010) the university with 
the highest enrollment in 2009 was the University of Phoenix, Online Campus, with an 
enrollment of 380,232 students. Allen & Seaman (2010) reported that in the fall of 2009 
an estimated 5.6 million students took an online course, an increase of one million 
students from the fall term of 2008. 
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As the popularity and convenience of online learning increases, so does the need for 
excellence and quality in the programs and courses offered. Becoming more scholarly in 
online delivery, especially in the evaluation or assessment of the methods used is 
essential for teaching and learning. Many universities use student evaluations to assess 
the quality of online courses. Student evaluations of the teaching strategies, materials 
and delivery methods used in online classes are essential for quality programs and can 
give instructors valuable information about their online course. Student evaluations that 
are conducted at the end of a semester can help faculty members adjust and alter their 
teaching methods. However, the dilemma is that end of semester student evaluations are 
too late to change strategies and delivery methods for the students currently enrolled. 
Also, changing methods for one class may not meet the expectations of another class. 
Students differ in each class based on their own expectations of the class, prior 
knowledge and technology experience. If feedback is given while the course is in 
session, online instructors are able to adjust strategies and methods to better meet the 
needs of current students. 

This paper explored the pilot of an evaluation of student learning through anonymous 
feedback at mid-semester in seven online courses. Electronic Formative Assessment of 
Classroom Teaching for Online Classes (eFACT) is a process of gathering anonymous 
student feedback through a faculty consultant using e-mail. This process gives the online 
instructor the opportunity to make changes to the delivery and quality of the course 
while the class is in session. 

LITERATURE REVIEW 

A kaleidoscope of viewpoints exists about online learning. The advantages of online 
courses cannot be denied. With an increasing adult student population, online programs 
offer flexibility and convenient learning opportunities (Sher, 2008). Online courses are 
convenient and reach out to students who would not be able to come to a face-to-face 
class because of family, job obligations, proximity and others issues. Online courses give 
students flexibility on learning the material and allowing them to learn it at their own 
pace. The advantages of online learning and its effectiveness in engaging student 
learners are numerous. Rohleder, Bozalek, Carolissen, Leibowitz & Swartz (2008) cite a 
study by Johns (2003) that attests to five virtues of online learning. 

> The course material is available at any time. 

> Students can spend more time on areas that are more difficult for them. 

> The Internet can bridge theory and real world experiences through 
different sites available online. 

> Online classes offer opportunities for students to be more active in their 
learning. 

> Students are engaged in higher-order learning skills like problem solving 
and gathering information. 

> 

In a study by Maki et al. (2000), young university students learned better in an online 
class than their counterparts in a face-to-face class. Students were given pre-and post¬ 
tests and the scores were twice as high for the online learners. 

This study was conducted over a number of semesters. However, the online learners 
consistently expressed less satisfaction in the course than the face-to-face learners (as 
cited in Moore & Kearsley, 2012). In a report by Simonson (2007), the U.S. Department 
of Education's Office of Postsecondary Education identified "Good Practices" and "Red 
Flags" in teaching and learning in online classes. For the purposes of this study, some 
positive indicators were: 


> The regular faculty are actively involved in course design; 

> There is 24/7 technology support; 

> Evaluation of distance education courses and programs are used for 
continuous improvement; and 

> Input from faculty and students are used for program improvement, (p. vii) 

Some red flags were identified as: 

> There are two course evaluation systems—one for traditional and one for 
distance education; 

> There are a large number of distant students who drop out; and 

> There are many complaints from distance students, (p. viii) 

Quality of the online course is also a concern. Liu (2005) also described two issues of 
online courses: quality and student persistence. She stated that the University of Florida 
in the fall of 1998 had withdrawal rates almost twice as high for online classes compared 
to the same course offered face-to-face. Adult learners are less likely to drop out when 
they are satisfied with the course and when it is relevant to their own lives (Park & Choi, 
2009). 

According to Keller (1987) allowing learners to choose learning strategies and methods 
bring satisfaction and relevance to online learning. "Online learners can easily lose 
motivation unless the course is designed to stimulate their active participation and 
interaction and meet their expectations" (as cited in Park & Choi, 2009, p. 215). 

There are more issues about online courses that persist about quality and perception. A 
number of studies have shown that even though online and face-to-face classes are 
comparable, online courses are still perceived as inferior (Pribesh, Dickinson, & Bucher, 
2006). Critics seem to hold online education to a higher standard than face-to-face 
classes (Tucker & Hodge, 2004). Tabs (2003) reports 26% of "US postsecondary schools 
feel that concerns about course quality are keeping them from either starting or 
expanding their distance education course offerings" (as cited in Liu, 2005, p. 361). 
Furthermore, Rohleder et al. (2008) report that the negative evaluations of online 
learning were: 

> Technical problems 

> Challenges of communicating effectively, and 

> A need for more face-to-face interactions 

According to a study by Hara and Kling (1999) students' frustrations with an online class 
were: 


> "(1) lack of prompt feedback from instructors, 

> (2) ambiguous instructions for assignments, and 

> (3) technical problems" (as cited in Moore & Kearsley, 2012, p. 164). 

In another study conducted by Frey, Alman, Barron and Steffens (2004) comments from 
students about a new online degree program were categorized into five issues: 

> program 

> course 

> communication 

> on-campus orientation, and 

> technology. 


121 



Overall, students expressed satisfaction about the program. Thirty-six students said they 
valued the program's convenience and flexibility and most participants expressed the 
degree would not be attainable in a face-to-face class. Students offered suggestions to 
improve the course. Some of their recommendations were to reduce student work-load, 
establish a consistent time when faculty would respond to email, provide more feedback 
and define a clear grading criteria. Students' comments concerning technology included 
a peer-support discussion board for questions and answers and faculty using the same 
style in the software management system for every course. 

Comfort with technology is a primary factor in determining satisfaction and success in an 
online course (Moore and Kearsely, 2012). The inability to use technology properly can 
affect students' experience in a negative way resulting in increased frustration and 
resistance to online classes. 

The ability to communicate effectively online is a concern for the instructor. Moore and 
Kearsley (2012) cite two challenges for the online instructor. One is that the instructor 
does not know how the students react to what they have written. "Students are 
generally more defensive when taking a course from an unseen instructor than they 
would be in a conventional class, but most are unlikely to express this anxiety" (p. 127). 
Another is that on-line teachers learn "on the job" and by "trial and error". According to 
Johnson and Brescia (2006) "Instructors need to receive focused training and be given 
opportunities to practice using the relevant technologies before they face students who 
are relying on them for instruction" (p. 71). 

Bangert (2006) states that quality online learning is in question as many universities and 
community colleges begin offering classes with little or no professional development for 
faculty as to best practices in online delivery. 

Other issues dealing with communicating in online classes are social isolation and a lack 
of community. Learning through online classes is both a social and individual process 
(Nevgi, Virtanin & Niemi, 2006). According to Abrami & Burnes (1996) social isolation 
contributes to weaknesses in online learning. They report online students experience few 
opportunities to interact with the instructor and other students to discuss assignments 
and concerns about their learning. Being physically separated may have an effect on 
their perception of being detached, which in turn, affects learning (as cited in Liu, 2005). 
Liu (2005) acknowledged the sense of community in a classroom that brings with it a 
feeling of belonging and trust is often nonexistent in an online class. She also affirmed 
that social intellectual isolation might be the biggest detriment to online class delivery. 
For online classes the challenge for many instructors becomes knowing about all the 
experiences, responsibilities, and learning styles of the students in the course (Johnson 
& Brescia, 2006). As a result the establishment of community, an essential part of 
students' experience in classroom learning may be difficult to establish for the online 
instructor. 

ASSESSMENT OF ONLINE CLASSES 

Assessment is essential for any online course and program. "Assessment is a key 
component of all educational programs; used properly it can yield valuable information 
for teachers, student and administrators" (Institute for Social Research, 2004, p. 1). 
There are summative assessments designed especially for online learning. Pike (2004) 
describes one type of summative assessment designed especially for online classes. SIR 
II from ETS measures eight dimensions and gives instructors feedback at the end of the 
semester. The eight measures include: 
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> Resources 

> Communication 

> Faculty-student interaction 

> Assignments, grading, and exams 

> Instructional methods and materials 

> Course outcomes 

> Student effort & involvement, and 

> Course difficulty, workload and pace 

Summative assessments, like the SIR II, are good measures of assessments for online 
classes and when taken at the end of the course give instructors important feedback. 
This feedback can be helpful for their next class but not for the students currently 
enrolled. 

Formative assessment, used extensively in K-12 classrooms, is given during the learning 
process and gives the teacher the opportunity to adjust and alter teaching practices. 
Formative assessment is used to improve student learning by improving the instructional 
practices of teachers (Dunn & Mulvenon, 2009). They conclude that research has 
demonstrated "that the use of formative assessment facilitates improvement in 
instructional practices, identifies 'gaps' in the curriculum, and contributes to increased 
student performance" (p.l). In an analysis of over 250 studies of formative assessment. 
Black & William (1998) revealed "feedback produced significant benefits in learning and 
achievement across all content areas, knowledge and skill types and levels of education" 
(as cited in Nicol & Macfarlane-Dick, 2006, p. 204). 

There are several definitions of formative assessment. Bloom, et al. (1971) described 
formative assessment as "the process of curriculum construction, teaching, and learning 
for the purpose of improving any of these three processes" (as cited in Pryor and 
Crossouard, 2008, p. 117). According to Popham (2008) formative assessment is a 
"planned process during which the teacher or students use assessment-based evidence 
to adjust ongoing learning and instruction" (as cited in Dunn and Mulvenon, 2009, p. 2). 

A formative assessment model created for higher education by Melnik and Allen at the 
University of Washington's Biology Learning Resource Center is called Small Groups 
Instructional Diagnosis (SGID) (Coffman, 1998). 

Coffman goes on to relate that SGID is a technique, also called "group interviews", for 
face-to-face classrooms where students use group consensus for feedback about what 
they like about the class. SGID uses open-ended questions usually at mid semester and 
is administered by a facilitator who asks students what they like about the course and 
what suggestions they have to improve it. 

This information is shared with the instructor giving them a chance to adjust their 
teaching mid-way through the course. 

eFACT was designed like the SGID to briefly shift the power to students at mid-semester 
to reflect on their own learning and give that feedback to the instructor so that he/she is 
able to change or alter methods and strategies to facilitate the learning process. 
According to Pryor and Crossouard (2008), an educator is also a learner so they need to 
make themselves vulnerable to student critiques. eFACT is a process where teachers as 
learners seek to understand the methods and strategies used in an online course that 
helps and/or hinders student learning. Going directly to the student for anonymous 
feedback at mid-semester is a powerful tool that can give online instructors detailed 
information concerning student learning. 



METHOD 


A pilot program began in the fall of 2010 for four instructors of seven online classes in a 
mid-western university in the United States to receive input about their course midway 
through the semester. Two of the classes were graduate level and five were 
undergraduate classes. To ensure that the information obtained was not used for 
evaluation of faculty, eFACT was administered by faculty for faculty. Faculty, already 
trained as consultants for Formative Assessment of Classroom Teaching designed for 
face-to-face classes, volunteered to be consultants for the new online eFACT. The eFACT 
was performed mid-semester after students had received a grade but with enough time 
in the semester left for potential changes. The process began when the consultant 
contacted the online instructor who had volunteered for the study by phone or e-mail. 
Together they set up a time-line, usually 5-7 days, for the collection of student data. 

The instructor added the consultant to the class as a student giving them full access to 
students' e-mails. This was accomplished through the university adopted course 
management system Blackboard. The instructor then e-mailed his/her students and let 
them know they would be contacted by e-mail from the consultant (naming the 
consultant) and encouraged them to reflect and respond to the questions about the 
online class. 

The consultant emailed the students outlining the procedure and assured students that 
communication with the consultant was confidential. The students were told that their 
instructor would get the compiled feedback and that all feedback would remain 
anonymous. The students were asked to spend about 10 minutes answering the 
following questions and then e-mail their responses to the consultant by a certain date, 
usually five days. The three questions were: 

> What helps your learning in this class? 

> What hinders your learning in this class? 

> What suggestions do you have for this class? 

Midway through the five days, the consultant emailed the class again encouraging them 
to respond to the first email. The consultant compiled the students' responses and e- 
mailed them to the students with instructions to read the list and make further 
comments, if needed. After collection of the data, the consultant emailed the instructor 
to delete them from the class. As with the SGID, after data had been collected and 
compiled the consultant met with the online instructor and gave him/her the compiled 
feedback. The consultant neither evaluated nor critiqued the faculty member or student 
feedback, but did instruct the faculty member to discuss the results with their students 
and thank them for their participation. The instructor then decided what, if any, changes 
needed to be made for increased student learning. 

FINDINGS 

In five online undergraduate classes and two graduate classes from three different 
colleges, instructors received useful information on student learning in their online 
classes. Fifty-four students responded to the e-mail. Most students were respectful and 
generous with their comments; however, some did take the opportunity to vent their 
frustrations. Students did not seem to have any apprehension about e-mailing the 
consultant their thoughts about their learning in the online class. Seven of the fifty-four 
students thanked the consultant in their e-mail for the ability to respond to the 
questions. 
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Technically speaking, everything in this pilot went as planned. However, in the seven 
online classes, there was no feedback from the students after emailing them the 
compiled responses. Even though this step is used successfully in face-to-face classes for 
consensus, it may not be a necessary step in online classes. 

Since research shows it is difficult to establish a sense of community in an online class 
because there are few if any face-to-face interactions, students may feel uncomfortable 
responding to other students' feedback. 

In accordance with the research of Johnson and Brescis (2006) most of the feedback 
centered on the online delivery format and not the discipline of the course. Only one 
student commented that the course discipline had always been a struggle for her. 

Student responses to what helped their learning centered around the following: 
availability of the instructor, detailed instructions for assignments, posting to the 
Discussion Board on Blackboard, audio-lectures, instruction on Power Point, Internet 
links and videos. 

Students' feedback included the following comments: 

> (The instructor was) very specific in what is expected, easy to contact and 
responds in a VERY timely fashion. 

> The discussion board has been very helpful along with the examples given. 

> The lectures and the Power Points helped me learn in this class. After 
reading the chapters it was nice to hear examples and watch interesting 
videos about the topics we were talking about to make the understanding 
of the material easier. 

> The notes and directions for class assignments are detailed and easy to 
follow. 

> (The instructor) was available by phone and email anytime I had 
questions. 

> I enjoy watching the videos, lectures, and reading the many articles about 
our topics instead of having a textbook. I think I learn better this way. 

> The audio streams help immensely in learning in this class. The links help 
to learn more about the issue. The discussion board helped to put what 
you learned into words. 

Students indicated that other tools that facilitated their learning were using Office Chat 
and posting of examples on Blackboard. Students commented about how online classes 
allowed them to proceed at their own pace that fit their lifestyle and helped their 
learning. One student commented; 

"I'm able to go at a pace that fits my lifestyle, for the most part..." 


Another student said; 

"Most of us take the online classes due to work schedu/es-in my 
case three jobs and I am a single mother." 

Hindrances to learning had more to do with the nature of online classes more than 
teaching strategies. Students' responses overwhelmingly talked about the lack of 
personal contact with classmates and teachers. 

> Lack of interaction with the teacher. No face-to-face interaction with 
teachers and classmates. 
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> I wish I had been able to take this class in class (face-to-face) instead of 
online. 

> I feel that it would be easier for me if I had personal contact with my 
teacher and classmates. 

> The hindrance in this class would be face-to-face time...there is not a 
replacement for actual face-to-face discussions. 

> The hindrance in this class would be the lack of face-to-face time. Though 
it is nice on the Internet because it fits in my schedule much easier, and 
the discussion board is a great place for bouncing off ideas, there is not 
replacement for actual face-to-face discussions. 

> The online nature of this course makes communication difficult at times. 

> Distance education is difficult in general. 

Other less common responses were technical problems that interfered with learning, 
using more than one textbook and not having lectures or test reviews. Changes in the 
syllabus were confusing to some students and having information spread out over the 
different sections of Blackboard made it difficult to locate all course materials and 
assignments. Time also was a hindrance as several students commented about the lack 
of time to put towards their work. This was especially true with the graduate classes. 
Students reported other responsibilities such as working full time and being a parent 
were hindrances to learning. 

> The only thing that I feel hinders my learning in class is finding adequate 
time to put towards my work due to working full-time as well. 

> Changes in the syllabus confuse me sometimes. 

> Having to look in numerous locations, different portals to see what is 
going on in class. 

> Nothing in the class hinders learning except for the computer problems 
I've had this semester. 

> Technology is fantastic but sometimes it gets in the way. 

Suggestions for the online classes included upgrading the Blackboard site, more time for 
homework and projects, detailed assignments and more communication from the 
instructor. Other individuals had concerns like retrieving an assignment in Podcast, 
having more assignments and less paper writing, providing a "hard" copy of the book, 
and giving students a voice in what is being learned. Sixteen students commented that 
no improvement was necessary or that they didn't have any suggestions at the time. 

All four instructors said the information they gained was useful in their teaching an 
online course. One instructor believed the eFACT was a way for students to provide 
meaningful information to the instructor. Another instructor commented, "Some of the 
responses involved wanting resources that were already available to the students. This 
shows the importance of repeatedly reminding students of all of the resources available. 
Some of the suggestions I could put into effect immediately. Other suggestions were not 
possible to implement, but it was important to explain to students the reasons why not. 
Each suggestion was addressed in the Blackboard announcements. Completing the 
eFACT was insightful and beneficial to the ongoing communication with the students." 
Instructors were able to make course changes as a result of the eFACT at mid-semester. 
They included: 


online assignments which were previously only listed online were included 
in the assignment sheet with the textbook assignments; 
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Additional instructor content was created for the course; tabs were created to give 
students direct access to course resources; and creating assignments with directions to 
optional resources so students knew where to find them when needed. But of equal 
importance was the continuation of the positive actions that instructors had already 
created for their courses. 


CONCLUSIONS 

eFACT is a power shift at mid-semester giving instructors the opportunity to adjust their 
teaching methods for the students enrolled in their class. Student learning in this study 
was helped by the use of email, videos, audio-lectures, office chat and discussion board. 

These features of online courses appear to make the class more social and interactive as 
in a face-to-face setting. Students may feel less isolated when they interact with the 
instructor and students through these tools. 

These interactions may give them a sense of community and a feeling of belonging both 
of which helps student learning. 

Most of the responses regarding hindrances to student learning had to do with perceived 
or real communication issues with the instructor and classmates. 


Students commented that face-to-face discussions were more valuable than discussion 
board. Although often citing the "nature" of the online class, the social aspect of 
learning seemed to be missing for many of the students. 

Also, students' knowledge of technology seemed to interfere with learning as students 
had difficulty finding assignments in more than one place in Blackboard and others 
mentioned technical/computer problems in general. Other hindrances included outside 
influences like a full time job and family. 

There are obvious challenges with online classes for the instructor and the student. As 
more and more classes and programs are offered online, quality of the delivery of those 
courses should be addressed. As more instructors accept responsibility for delivering 
online courses, new instruments and assessments need to be developed to give 
instructors accurate feedback about their teaching methods (Bangert, 2006). Open- 
ended questions at mid-semester can help elicit detailed perceptions of student learning. 
Assessment measures like eFACT can affect the quality of the online course by giving 
instructors immediate feedback as students reflect on their learning in the class and 
allowing the instructors to make changes right away. 
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